docs: propose workload-wide planning, summary sharing, and materialization - #509

Describe the two plain candidate collections, the view-based compile, the
caller-typed physical candidate errors, lifecycle_guarantee, and
PhysicalExecution as an execution handle. Remove APIs the docs said were
removed but never existed (compile_timed_candidates, PhysicalDAGCandidate),
the agent instructions in the alignment proposal's baseline, and the
ingestion-time Binary exception from design docs, where it is an
implementation detail (the developer migration guide keeps it).

Restore #485's statement that candidates do not choose placement and #508's
CandidatePostASAPDAGs<Id> names in input-output-workflow.md, rejoin the
split test table in physical-planning-and-deployment.md, and take
planner-backend-layering.md verbatim from #509.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Copies docs/design_docs/proposals/planner-backend-layering.md from #509's
head 3607270 so this file merges cleanly after #509.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The layering proposal (#509) names a single annotated DAG,
LifecyclePostASAPDAG: a PostASAPDAG plus its lifecycle assignment. The
code kept a SummaryMaintenanceLifecyclePlan beside the DAG and named the
timed collection CandidatePostASAPDAGsWithTiming.

- Rename SummaryMaintenanceLifecyclePlan to LifecyclePostASAPDAG and its
  error to LifecyclePostASAPDAGError. The type already held the root and
  each state's lifecycle, retention and window framework; per-node timing
  stays derived by execution_assignment as the existing
  PostASAPDAGAssignment overlay, so no timed graph is stored beside it and
  no new type is added.
- Rename CandidatePostASAPDAGsWithTiming to CandidateLifecyclePostASAPDAGs;
  CandidatePostASAPDAGs stays the logical collection.
- Call layer 2 "summary lifecycle planning" and reword comments that had
  the deployment choose or rank; selection is Planner's, over the
  deployment's cost model.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Several docs still said the deployment or "downstream" selects, ranks or
chooses the plan, and the glossary said physical plans are owned by
downstream systems. Per the layering proposal (#509), Planner compiles
and selects the physical plan using the deployment's cost model; the
deployment supplies prices and executes the selected plan.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

#509 carries the proposal; #480's copy now matches its head a095ce5
exactly, so merging either PR first leaves no conflict.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…tive weights

DataWorkload gains `metric_types` (metric name -> Counter | Gauge, the
Prometheus TYPE metadata). plan_stages passes it to Stage 1, and Pass 1
attaches the new `CounterSamples` non-negativity proof to a keyed summary
update weighted by the sample value when the target's input is samples of a
declared counter, through a time range and plain sums (sum_over_time). Any
other operator, a gauge or an undeclared metric (whatever its suffix) gets
no proof. Stage 3 and the runtime already accept any NonNegative proof for
Count-Min, so their rule is unchanged.

Example 1 declares http_requests_total a counter: all 64 candidates are
valid and compile (24 Count-Min + heap were invalid). Stage 3 now selects
P60 (Count-Min + heap over the exact sum_over_time accumulator, shared
input) at 46.201 against P58 (all exact) at 52.201, now 8th.

Refs #509, #580.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…umers

The summary-capability rule gains a frequency-moments key: a distinct count
(one column), an L2 norm and an entropy over the same input and window
share one UnivMon (#509 Example 2). UnivMon's shape does not depend on the
requirement, so the three alternatives are the same state.

The UnivMon accuracy model still certifies only the exact total. The
published UnivMon bounds (Liu et al., SIGCOMM 2016, via Braverman and
Ostrovsky) are asymptotic. The shipped kernel's heuristic recurrence, a
fixed top-heap per layer with an L2/sqrt(heap) cut, does not meet their
per-layer cover premise. Distinct count, L2 and entropy therefore stay
uncertified. The reasoning is documented, and a test pins it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…ntities

Port of the parked runtime PRs for #509 Example 2 onto asap-executor:

- native UnivMon build and Cardinality / FrequencyL2 / FrequencyEntropy
  readouts (the old stack's 14c8ac8, a prerequisite missing here);
- #557: a UnivMon build accepts Utf8, Int64 and Bool identities as
  type-tagged keys; heap key bytes count toward state memory;
- #559: exact FrequencyL2 / FrequencyEntropy reducers (bits, NULL skipped,
  0 for an empty population, memory-accounted, cooperative);
- #563: exact Cardinality reducer over typed tuples.

The legacy raw-cost adapter lists the three intents as hash aggregates.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Port of the parked #562 and #564 for #509 Example 2. The SQL frontend
names `SQRT(SUM(c*c))` over Float64 grouped unit counts as FrequencyL2,
and `-SUM(p*LN(p))` with `p = COUNT(*)*1.0 / SUM(COUNT(*)) OVER ()` as
FrequencyEntropy converted to nats. An exact population count guards
SQL's empty-input NULL. Integer products, nullable keys, filters, HAVING,
other log bases and partial windows are refused.

The rules are the old Pass 1 SemanticEquivalentRewriteStrategy rules, run
by lower_sql at the query root so the Stage 1 pipeline sees the intents;
the exact candidate is the native exact reducer. The executor gains SQL
sqrt (from #562).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Port of the parked #565: SQL LN and a complete, unordered
`SUM(column) OVER ()` window (SQLWindowSum) run natively, so the original
Q2 SQL of #509 Example 2 executes and matches its recognized entropy form.
Partitioned, ordered and finite frames stay refused.

planner_layering_example2 records Example 2's status: the three queries
lower to Cardinality / FrequencyEntropy / FrequencyL2 (the design's integer
Q3 does not), the time filter is a scan predicate, plan_stages plans the
workload with UnivMon offered to each statistic, and the exact candidates
execute (without the time filter: the runtime has no now() yet).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Load an asap-stage-pipeline/v1 document (file picker or ?doc=<path>) and
show Logical, Logical ASAP and Physical ASAP lanes side by side, with
candidate switchers, per-node timing and cost, a cost-ranked list of
physical candidates, and the stage-3 selection and rejection reasons.
Selecting a physical candidate shows its from_logical candidate in lane 2.
Pre/Post-ASAP stays the default unless a stage document is loaded.

stages.js holds the DOM-free validation, lane construction and ranking,
tested headless against a hand-written #509 Example 1 Q2 fixture.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Contract v2 (Q28-Q30) gives each DAG one root per workload query and
moves cost out of Stage 2 into stage3_selection.costs.

- Read `roots` (still accepting a single `root`); mark every root and
  label it with its query id.
- Take node badges, totals and ranking only from Stage 3 costs, labelled
  as a Stage 3 result; without Stage 3 the physical lane has no cost.
- Load partial documents; missing later stages render "not produced".
- Require every Stage 2 candidate to be selected or rejected, and show
  valid-but-costlier apart from invalid.
- Show optional per-query requirements in the query list and on roots.
- Rewrite the sample as #509 Example 1 with both queries, and label
  sort/limit physical operators.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

stage_pipeline now prices every plan and writes the cheapest
--max-candidates (#613); the Stages view says "showing N of M plans" from
the document's shown_of section. The README shows how to generate all six
#509 examples into an ignored out/ directory.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Pre/post-ASAP DAGs no longer exist after #509/#572, so the viewer now
reads only asap-stage-pipeline/v1 documents and uses the layout of the
published Stage Viewer: example tabs with what each shows, the workload's
queries and the deployment inputs on top, the Stage 3 plans ranked by cost
beside three DAG lanes (Stage 0, the chosen Stage 1 candidate, the chosen
Stage 2 candidate with costs), and node and edge details below.

- index.html and app.js replace viewer.js; stages.js is unchanged and
  node-style.js keeps only the kind table the contract test reads, plus
  the three node groups.
- examples.json lists the six #509 examples; server.py writes any
  missing document into out/ with stage_pipeline.
- The query editor (editor.js) plans PromQL queries with optional ε/δ
  through /api/plan, which now runs stage_pipeline.
- Removed: the Pre/Post-ASAP view, WorkloadDAG loading, render.py's
  standalone HTML, the dag_export-based sample and fixtures, screenshots.
- test_viewer.py replaces test_render.py: the stage-document tests, the
  page and server checks, and app.js run against a stub DOM in V8.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Library API, architecture overview, planner pipeline and the accuracy
and cost developer docs now describe the #509 stage pipeline. Docs
that only describe removed APIs (legacy search, CostModel,
explanation, physical-plan cost adapter) are marked historical.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>